61 research outputs found
Multimodal Subspace Support Vector Data Description
In this paper, we propose a novel method for projecting data from multiple
modalities to a new subspace optimized for one-class classification. The
proposed method iteratively transforms the data from the original feature space
of each modality to a new common feature space along with finding a joint
compact description of data coming from all the modalities. For data in each
modality, we define a separate transformation to map the data from the
corresponding feature space to the new optimized subspace by exploiting the
available information from the class of interest only. We also propose
different regularization strategies for the proposed method and provide both
linear and non-linear formulations. The proposed Multimodal Subspace Support
Vector Data Description outperforms all the competing methods using data from a
single modality or fusing data from all modalities in four out of five
datasets.Comment: 26 pages manuscript (6 tables, 2 figures), 24 pages supplementary
material (27 tables, 10 figures). The manuscript and supplementary material
are combined as a single .pdf (50 pages) fil
Boosting rare benthic macroinvertebrates taxa identification with one-class classification
Insect monitoring is crucial for understanding the consequences of rapid
ecological changes, but taxa identification currently requires tedious manual
expert work and cannot be scaled-up efficiently. Deep convolutional neural
networks (CNNs), provide a viable way to significantly increase the
biomonitoring volumes. However, taxa abundances are typically very imbalanced
and the amounts of training images for the rarest classes are simply too low
for deep CNNs. As a result, the samples from the rare classes are often
completely missed, while detecting them has biological importance. In this
paper, we propose combining the trained deep CNN with one-class classifiers to
improve the rare species identification. One-class classification models are
traditionally trained with much fewer samples and they can provide a mechanism
to indicate samples potentially belonging to the rare classes for human
inspection. Our experiments confirm that the proposed approach may indeed
support moving towards partial automation of the taxa identification task.Comment: 5 pages, 1 figure, 2 table
Multidimensional Particle Swarm Optimization for Machine Learning
Particle Swarm Optimization (PSO) is a stochastic nature-inspired optimization method. It has been successfully used in several application domains since it was introduced in 1995. It has been especially successful when applied to complicated multimodal problems, where simpler optimization methods, e.g., gradient descent, are not able to find satisfactory results. Multidimensional Particle Swarm Optimization (MD-PSO) and Fractional Global Best Formation (FGBF) are extensions of the basic PSO. MD-PSO allows searching for an optimum also when the solution dimensionality is unknown. With a dedicated dimensional PSO process, MD-PSO can search for optimal solution dimensionality. An interleaved positional PSO process simultaneously searches for the optimal solution in that dimensionality. Both the basic PSO and its multidimensional extension MD-PSO are susceptible to premature convergence. FGBF is a plug-in to (MD-)PSO that can help avoid premature convergence and find desired solutions faster. This thesis focuses on applications of MD-PSO and FGBF in different machine learning tasks.Multiswarm versions of MD-PSO and FGBF are introduced to perform dynamic optimization tasks. In dynamic optimization, the search space slowly changes. The locations of optima move and a former local optimum may transform into a global optimum and vice versa. We exploit multiple swarms to track different optima.In order to apply MD-PSO for clustering tasks, two key questions need to be answered: 1) How to encode the particles to represent different data partitions? 2) How to evaluate the fitness of the particles to evaluate the quality of the solutions proposed by the particle positions? The second question is considered especially carefully in this thesis. An extensive comparison of Clustering Validity Indices (CVIs) commonly used as fitness functions in Particle Swarm Clustering (PSC) is conducted. Furthermore, a novel approach to carry out fitness evaluation, namely Fitness Evaluation with Computational Centroids (FECC) is introduced. FECC gives the same fitness to any particle positions that lead to the same data partition. Therefore, it may save some computational efforts and, above all, it can significantly improve the results obtained by using any of the best performing CVIs as the PSC fitness function.MD-PSO can also be used to evolve different neural networks. The results of training Multilayer Perceptrons (MLPs) using the common Backpropagation (BP) algorithm and a global technique based on PSO are compared. The pros and cons of BP and (MD-)PSO in MLP training are discussed. For training Radial Basis Function Neural Networks (RBFNNs), a novel technique based on class-specific clustering of the training samples is introduced. The proposed approach is compared to the common input and input-output clustering approaches and the benefits of using the class-specific approach are experimentally demonstrated. With the class-specific approach, the training complexity is reduced, while the classification performance of the trained RBFNNs may be improved.Collective Network of Binary Classifiers (CNBC) is an evolutionary semantic classifier consisting of several Networks of Binary Classifiers (NBCs) trained to recognize a certain semantic class. NBCs in turn consist of several Binary Classifiers (BCs), which are trained for a certain feature type. Thanks to its topology and the use of MD-PSO as its evolution technique, incremental training can be easily applied to add new training items, classes, and/or features.In feature synthesis, the objective is to exploit ground truth information to transform the original low-level features into more discriminative ones. To learn an efficient synthesis for a dataset, only a fraction of the data needs to be labeled. The learned synthesis can then be applied on unlabeled data to improve classification or retrieval results. In this thesis, two different feature synthesis techniques are introduced. In the first one, MD-PSO is directly used to find proper arithmetic operations to be applied on the elements of the original low-level feature vectors. In the second approach, feature synthesis is carried out using one-against-all perceptrons. In the latter technique, the best results were obtained when MD-PSO was used to train the perceptrons.In all the mentioned applications excluding MLP training, MD-PSO is used together with FGBF. Overall, MD-PSO and FGBF are indeed versatile tools in machine learning. However, computational limitations constrain their use in currently emerging machine learning systems operating on Big Data. Therefore, in the future, it is necessary to divide complex tasks into smaller subproblems and to conquer the large problems via solving the subproblems where the use of MD-PSO and FGBF becomes feasible. Several applications discussed in this thesis already exploit the divide-and-conquer operation model
Color Constancy Convolutional Autoencoder
In this paper, we study the importance of pre-training for the generalization
capability in the color constancy problem. We propose two novel approaches
based on convolutional autoencoders: an unsupervised pre-training algorithm
using a fine-tuned encoder and a semi-supervised pre-training algorithm using a
novel composite-loss function. This enables us to solve the data scarcity
problem and achieve competitive, to the state-of-the-art, results while
requiring much fewer parameters on ColorChecker RECommended dataset. We further
study the over-fitting phenomenon on the recently introduced version of
INTEL-TUT Dataset for Camera Invariant Color Constancy Research, which has both
field and non-field scenes acquired by three different camera models.Comment: 6 pages, 1 figure, 3 table
Neural Architecture Search by Estimation of Network Structure Distributions
The influence of deep learning is continuously expanding across different
domains, and its new applications are ubiquitous. The question of neural
network design thus increases in importance, as traditional empirical
approaches are reaching their limits. Manual design of network architectures
from scratch relies heavily on trial and error, while using existing pretrained
models can introduce redundancies or vulnerabilities. Automated neural
architecture design is able to overcome these problems, but the most successful
algorithms operate on significantly constrained design spaces, assuming the
target network to consist of identical repeating blocks. While such approach
allows for faster search, it does so at the cost of expressivity. We instead
propose an alternative probabilistic representation of a whole neural network
structure under the assumption of independence between layer types. Our matrix
of probabilities is equivalent to the population of models, but allows for
discovery of structural irregularities, while being simple to interpret and
analyze. We construct an architecture search algorithm, inspired by the
estimation of distribution algorithms, to take advantage of this
representation. The probability matrix is tuned towards generating
high-performance models by repeatedly sampling the architectures and evaluating
the corresponding networks, while gradually increasing the model depth. Our
algorithm is shown to discover non-regular models which cannot be expressed via
blocks, but are competitive both in accuracy and computational cost, while not
utilizing complex dataflows or advanced training techniques, as well as
remaining conceptually simple and highly extensible.Comment: 16 pages, 4 figures, 3 table
On Feature Diversity in Energy-based Models
Energy-based learning is a powerful learning paradigm that encapsulates
various discriminative and generative approaches. An energy-based model (EBM)
is typically formed of inner-model(s) that learn a combination of the different
features to generate an energy mapping for each input configuration. In this
paper, we focus on the diversity of the produced feature set. We extend the
probably approximately correct (PAC) theory of EBMs and analyze the effect of
redundancy reduction on the performance of EBMs. We derive generalization
bounds for various learning contexts, i.e., regression, classification, and
implicit regression, with different energy functions and we show that indeed
reducing redundancy of the feature set can consistently decrease the gap
between the true and empirical expectation of the energy and boosts the
performance of the model.Comment: 18 pages, 3 figure
- …